METHOD AND APPARATUS FOR DETECTING OBJECTS 



in 



This application claims priority under 35 U.S.C.§ 119(e)(1) to co-pending U.S. 
Provisional Patent Application Serial No. 60/262,925, filed January 19, 2001, and entitled 
5 "OBJECT DETECTION USING MOIRE INTERFERENCE". 

Field of the Invention 
The present invention relates to object detection, and more specifically, to object 
intrusion and/or presence detection within a monitored area or region. 

Background of the Invention 
O 10 Motion detection and object detection systems are well known in the art. 

:iS Frequently, such systems monitor a user-defined area to detect when an object enters or 

passes through a monitored area. Such systems typically include an image capture device 
(typically a video camera or still camera) capable of capturing an image of the monitored 
area and, if required, a device for digitizing the captured images. The digitized images are 
15 analyzed in an attempt to detect whether an object has entered the monitored area. There 
are many different known methods and algorithms for analyzing digitized images for 
determining when an object has entered a monitored area. One of the most common 
methods is generally referred to as a change detection method. 

Change detection is often accomplished by examining the difference between a 
20 current live image and a reference image, where the reference image contains only the 
static background of the monitored area. A reference image can be thought of as a 
representation of the monitored area as it would appear if no transitory objects were in 
view. Change detection algorithms often take two digitized images as input and return the 
locations in the field of view where differences between the images are identified. 
25 Object detection systems are commonly used in environments that have dynamic 

lighting conditions. For example, in industrial settings, moving shadows can be cast on a 
monitored area or region, which can cause significant changes in ambient lighting 
conditions. Many existing object detection systems, including those that use change 



detection algorithms to detect objects, can be challenged by such shadows and/or other 
dynamic lighting conditions. 

Summary of the Invention 
The present invention overcomes many of the disadvantages of the prior art by 
5 providing an object detection system that is less susceptible to dynamic lighting 

conditions, and/or more sensitive to three-dimensional object motion and/or presence. 
This is preferably accomplished by projecting one or more static or dynamic patterns 
onto the monitored area, capturing one or more live images of the monitored area 
including the one or more patterns, and detecting objects in the monitored area by 
10 detecting changes in the one or more patterns in selected captured images. 

In a first illustrative embodiment of the present invention, a single pattern is 
projected onto the monitored area. One or more live images of the monitored area are 
then captured at selected times, and analyzed to detect changes in the projected pattern. 
The changes in the pattern may indicate a topographical change in the monitored area, 
15 and thus the entry or movement of an object in the monitored area. Because the pattern is 
projected onto the monitored area, changes in the ambient lighting conditions may have 
less effect on the efficacy of the object detection system. 

In another illustrative embodiment, a moire interference pattern is used to help 
detect objects in the monitored area. Moire interference patterns are particularly 
20 sensitive to relative motion between the two or more underlying patterns that are used to 
create the moire interference pattern. As such, the use of moire interference patterns can 
be highly effective in detecting objects that intrude into a monitored area. 

A moire interference pattern may be created in any number of ways. For 
example, two or more similar patterns may be projected onto the monitored area from 
25 offset illumination positions. Alternatively, or in addition, one pattern may be projected 
onto the monitored area, while another may be imposed by a patterned grating positioned 
in the image plane of the image capture device. Yet another way of creating a moire 
interference pattern is to capture two images of the same area having a single projected 
pattern, and digitally or optically rotating or otherwise changing the position of one of the 



images relative to the other to create a moire interference pattern. Any number of other 
methods may also be used to create a moire interference pattern, as desired. 

When using a moire interference pattern, one or more live images of the 
monitored may be captured at selected times. The images may then be analyzed to detect 
changes in the moire interference pattern. Changes in the moire interference pattern may 
indicate a topographical change in the monitored area and thus the entry or movement of 
an object in the monitored area. An advantage of using moire interference patterns is that 
shadows and/or other changes in ambient lighting conditions may have little or no effect 
on the position, frequency or other characteristics of the moire interference pattern. To 
help prevent the projected pattern from being overwhelmed by ambient light, it is 
contemplated that a portion of the spectrum or lighting frequency that is not typically 
found in the ambient lighting conditions may be used to project the one or more patterns 
on the monitored area, such as near infrared. 

The present invention may be used to monitor a user-defined safety zone for the 
intrusion of people or other objects. However, numerous other applications are also 
contemplated including security, recording, and other monitoring and/or detection 
applications. 

Brief Description of Drawing s 

Figures 1 A-1B are schematic diagrams showing one illustrative object detection 
system in accordance with the present invention; 

Figures 2A-2B are schematic diagrams showing other illustrative object detection 
systems in accordance with the present invention; 

Figures 3 A-3B depict two example patterns that can be used in accordance with 
some embodiments of the present invention; 

Figures 4A-4B depict two examples of patterns that can be used in accordance 
with other embodiments of the present invention; 

Figures 5 A-5C depict an example of moire interference phenomena in accordance 
with the present invention; 



Figures 6A-6C depict an illustrative reference image, live image and comparison 
image, respectively, in accordance with one embodiment of the present invention; 

Figure 7 is a flow diagram showing an illustrative method in accordance with 
present invention; 

Figure 8 is a flow diagram showing another illustrative method in accordance 
with the present invention; and 

Figure 9 is a flow diagram showing yet another illustrative method in accordance 
with the present invention. 

Detailed Description of the Invention 
The following description should be read with reference to the drawings wherein 
like reference numerals indicate like elements throughout the several drawings. The 
detailed description and drawings represent select illustrative embodiments and are not 
intended to be limiting. 

The present invention provides an object detection system that may be less 
susceptible to dynamic lighting conditions, and/or may be more sensitive to object 
motion and/or presence than prior art systems. The system is also sensitive to all three- 
dimensional extraneous objects that intrude into the monitored area. This may be 
accomplished by, for example, projecting one or more static or dynamic patterns on the 
monitored area, capturing one or more images of the monitored area including the one or 
more patterns, and detecting objects in the monitored area by detecting changes in the 
one or more patterns in selected captured images. 

Figures 1A-1B are schematic diagrams showing a first illustrative object detection 
system in accordance with the present invention. The illustrative object detection system 
of Figures 1A includes an illumination source 2, an image capture device 4, an image 
storage device 6, and a processing device 8. In some embodiments, the illumination 
source 2, image capture device 4, image storage device 6, and processing device 8 are 
integrated into a common device, while in other embodiments, separate devices are 
provided, as desired. 

The illumination source 2 is located above a monitored area 14, such as near a 
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ceiling. The illumination source 2 illuminates the monitored area 14 with a desired 
pattern. The pattern may be generated by, for example, projecting through a patterned 
grating, projecting interference patterns where the interference fringes are formed by a 
phasor or wavelength shifting, projecting a pattern using a scanning mechanism, or any 
other suitable method. 

As noted above, the pattern may be static or dynamic. A dynamic pattern is one 
where the spatial position of the light areas and dark areas is moving, and in general the 
movement is periodic in nature. One way to realize a dynamic moire pattern is to project 
an interference pattern from a Mach-Zender interferometer, which may include a mirror 
on a peso-actuator. The frequency of movement of the mirror may determine the 
frequency of interference fringe movement. It is contemplated that the monitored area 14 
may be rectangular, round, or any other shape, as desired. As shown in Figure IB, the 
illustrative monitored area 14 extends around three side of a machine 18. 

It is contemplated that the illumination source 2 may be any type of illumination 
source capable of projecting a desired pattern on the monitored area 14. Examples of 
suitable illumination sources 2 include an incandescent or fluorescent lamp with a 
patterned mask and illumination optics. Other examples of suitable illumination sources 2 
include a lasing source such as a modulated light-stripe illumination module, or a laser 
diode source uniformly illuminating a patterned grating with the grating re-imaged onto 
the monitored area with an objective projection lens. Yet another example of a suitable 
illumination source 2 may be an infrared source. Using a portion of the spectrum not 
ordinarily found in the ambient lighting conditions, such as the near infrared, may help 
keep the projected pattern from being overwhelmed by the ambient lighting conditions, 
and may also help enhance the differentiation between the projected pattern and other 
sources of light in the environment. In addition, or alternatively, the image capture 
device may be a sensor (CCD or the like) that is attuned to a desired spectrum, such as 
the spectrum of the illumination source. 

The illumination source 2 preferably projects at least one pattern on the monitored 
area 14. The pattern used may depend on the particular application at hand. For example, 
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in one illustrative embodiment, the pattern may be any pattern that has transitions 
between areas that have illumination (e.g. light areas) and areas that lack illumination 
(e.g. dark areas). Generally, the average distance between transitions should be 
approximately the same size as the smallest object for which detection is desired, 
although this is not required. Examples of suitable patterns included striped or checker 
board patterns where illuminated and non-illuminated areas alternate. Some illustrative 
patterns are shown in Figures 3 A-3B and Figures 4A-4B, but any suitable pattern may be 
used. 

Once illuminated, and in one illustrative embodiment, the image capture device 4 
captures a reference image of the monitored area 14. The reference image is preferably 
an image of the monitored area 14 with no transitory objects in the monitored area 14. 
The reference image is preferably stored, at least temporarily, in the image storage device 
6. A new reference image may be captured periodically, if desired. Once a reference 
image is captured, the image capture device 4 may capture successive live images of the 
monitored area 14, preferably at a selected time interval. Like the reference image, the 
live images are preferably stored, at least temporarily, in the image storage device 6. The 
image storage device 6 may provide the reference image and the live images to the 
processing device 8 for processing. 

The processing device 8 preferably analyzes the live images to detect changes in 
the illuminated pattern. In some embodiments, the monitored area is divided into a 
number of image segments called mask windows. The size of each mask window is 
preferably chosen so that it is no bigger than the approximate size of the smallest object 
for which detection is desired. While objects smaller than the mask window may be 
detected, the probability of detecting such objects decreases with object size. The 
position of the various mask windows may be chosen so that the entire area to be 
monitored is covered by overlapping mask windows. The image area that corresponds to 
each mask window may be analyzed separately for object detection, if desired. In 
addition, it is contemplated that the analysis method that is used to analyze the various 
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mask windows may differ across the image, and the triggered response may vary 
depending on which mask window detects an object, if desired. 

The comparison between a reference image and a live image can be accomplished 
in any number of ways. One method is to simply do a pixel-by-pixel comparison of the 
images, such as by subtracting one image from the other. If there is no entry or 
movement of objects in the monitored area, the projected pattern in the two images will 
substantially cancel out. However, if there is entry or movement of an object in the 
monitored area, part of the projected pattern shown in one image may be shifted or 
otherwise deformed relative to the pattern shown in the other image. A threshold value 
may be used to help determine if there is a sufficient difference between the reference 
image and a live image to indicate a detected object, as further described below. 

Another method for comparing one image to another is to calculate a difference 
"gref T between the value of the brightness levels corresponding to the light areas of the 
pattern (such as in a mask window), and the value of the brightness levels corresponding 
to the dark areas in the mask window of the reference image. A similar calculation may 
be made for the mask windows of a live image. Whenever the second calculation is 
different from the first calculation by a specified amount, a change may be inferred. A 
further discussion of this and other methods for comparing images is included in co- 
pending U.S. Patent Application Serial No. , entitled "Object 

Detection", which is incorporated herein by reference. 

Yet another method for comparing one image to another is to measure a correlation 
between each pixel and some neighboring pixels and/or a correlation between selected 
features, and then compare the correlation values. Whenever the correlation values are 
different by a specified amount, a change may be inferred. Alternatively, or in addition, 
the image analysis may extract the moire spatial frequency and phase using a Fourier 
transform. Other image analysis techniques may also be used including, for example, 
unsharp masking, thresholding, contrast segmentation, filtering processing, 
skeletonization processing, multi-resolution analysis, deformable contour modeling, 
image clustering, morphology, etc. These comparison methods are meant to be only 
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illustrative, and that any suitable method may be used to compare the images or selected 
characteristics of the images, depending on the application. 

It is also contemplated that the reference and/or live images may be preprocessed 
before they are compared. For example, the reference image and/or live images may be 
provided to a filter that helps removes speckle, provides smoothing, changes overall 
intensity, or otherwise cleans-up the images. 

Changes that are detected in the illuminated pattern may indicate a topographical 
change within the monitored area 14, and thus entry or movement of an object in the 
monitored area 14. When an object is detected, the processing device 8 may sound an 
alarm, shut down the machine 18, and/or provide some other alarm or action. Images of 
the monitored area with the detected object present may be retained for subsequent 
analysis, and/or sent to a monitoring station if desired. 

Figures 2A-2B are schematic diagrams showing yet other illustrative object 
detection systems in accordance with the present invention. In Figure 2A, the 
illumination source 2 includes a radiation source 20, an objective projection lens 22, and 
a patterned grating 24. In the illustrative embodiment, the patterned grating 24 provides 
the desired pattern on the monitored area 14. It is contemplated that any pattern suitable 
for creating a moire interference pattern may be used. Some suitable patterns are shown 
in Figures 4A-4B. Rather than providing a separate patterned grating, the illumination 
source 2 itself may be configured to provide the desired pattern, if desired. 

In the illustrative embodiment shown in Figure 2A, the image capture device 4 
captures a first image of the monitored area 14 through a second grating 26. The second 
grating 26 may be any type of grating, filter or mask that produces the desired pattern. In 
some cases, the second grating 26 may be provided by the CCD pixel array of the image 
capture device 4 itself. The first grating 24 and the second grating 26 preferably 
collectively produce a live image that including a moire interference pattern therein. In 
some embodiments, the second pattern may be imposed digitally on the live image after 
the live image has been captured, if desired. The image capture device 4 transmits the 
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live image to an image storage device 28, which in the embodiment shown, may also 
include a processing device. 

The position of the illumination source 2 and the image capture device 4 are 
preferably spaced from each other by a distance. This distance can be small to large. 
This space may help encourage a change in the pattern that reaches the image capture 
device 4 when a topographical change occurs in the monitored area 14. Typically, the 
space should be larger as the resolution of the projected pattern is increased. Finally, a 
warning device 30 may be attached to the image storage device 28, and may be activated 
upon detection of an object. 

In Figure 2B, two illumination sources 36 and 38 are provided for projecting two 
separate patterns on the monitored area 14. Illumination source 36 includes a radiation 
source 40, an objective projection lens 42, and a patterned grating 44. The illumination 
source 36 preferably projects a first pattern on the monitored area 14. Likewise, 
illumination source 38 includes a radiation source 46, an objective projection lens 48, and 
a patterned grating 50. Illumination source 38 preferably projects a second pattern on the 
monitored area 14. The first pattern and the second pattern preferably collectively cause 
a moire interference pattern to be formed on the monitored area 14. 

Image capture device 4 is used to capture a live image of the monitored area 14. 
Like above, the image capture device 4 may include an image storage device and a 
processing device. The live images are preferably analyzed to detect changes in the moire 
interference pattern over time. Changes in the moire interference pattern may indicate a 
topographical change in the monitored area 14 and thus the entry or movement of an 
object in the monitored area 14. 

The illustrative embodiments shown in Figures 2A-2B both cause a moire 
interference pattern to be provided in the resulting live image. It is contemplated that the 
image analysis may be similar to that discussed above with respect to Figures 1 A-1B. 
Alternatively, or in addition, the image analysis may extract the moire spatial frequency 
and phase using a Fourier transform. Other image analysis techniques may also be used 
including, for example, unsharp masking, thresholding, contrast segmentation, filtering 
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processing, skeletonization processing, multi-resolution analysis, deformable contour 
modeling, image clustering, morphology, etc. Alternatively, or in addition, a radon filter 
may be oriented perpendicular to the moire interference bands, and any loss of correlation 
between the filtered results from the reference image and the live image may indicate a 
change in the moire interference bands. 

Figures 5A-5C depict an example of moire interference phenomena in accordance 
with the present invention. Figure 5 A shows an image of a first pattern projected onto a 
monitored area. The first pattern includes a number of parallel lines extending in the 
vertical direction. Figure 5B shows an image of the monitored area with a second pattern 
superimposed on the first pattern. The second pattern includes a number of parallel lines 
extending in a direction that is radially offset relative to the vertical direction. In the 
illustrative embodiment shown, the surface of the monitored area is spherical in shape, 
resulting in a number of curved moire interference bands 50. Figure 5C shows an image 
of the monitored area with an object 52 positioned in the monitored area. The object 52 
causes a change in the moire interference bands relative to Figure 5B. As indicated 
above, the moire interference bands are highly sensitive to topographical changes in the 
monitored area, which in the embodiment shown, is illustrated by the introduction of the 
three-dimensional object 52 into the monitored area. 

Figures 6A-6C depict an illustrative reference image, live image and comparison 
image, respectively, where only a single pattern is projected onto the monitored area and 
no grating is positioned between the monitored area and the image capture device. The 
image shown in Figure 6A is a reference image of the monitored area. The image shown 
in Figure 6B is an image of the monitored area when a hand is placed in the monitored 
area. The image shown in Figure 6C is an image that results from the comparison (e.g. 
subtraction) of the image shown in Figure 6 A and the image shown in Figure 6B. As can 
be seen, the image shown in Figure 6C highlights the object, including its boundary, 
within the monitored area. In addition, virtual interference bands appear in and around 
the object. Under some circumstances, it may be desirable to provide a tolerance or 
threshold below which differences are likely due only to normal variations in lighting 
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conditions or between images taken of the same area at different times. This tolerance 
may reduce the likelihood of false positives, and increase the reliability of the system. 

Figure 7 is a flow diagram showing an illustrative method in accordance with 
present invention. The illustrative method is entered at step 70, wherein a threshold value 
is selected. Control is then passed to step 72. Step 72 illuminates a monitored area with 
a pattern. As described above, the pattern may be any suitable pattern. Step 74 then 
captures a reference image of the monitored area. The reference image preferably 
contains only the static background of the monitored area, with no transitory objects 
present. In some embodiments, the reference image can be captured/updated upon 
manual initiation, as shown at 88. 

Step 76 then captures a live image of the monitored area. Step 78 compares the 
reference image and the live image to determine a difference parameter. As detailed 
above, the comparison of the reference image and the live image can be accomplished in 
any number of ways. One method is to simply do a pixel-by-pixel comparison of the 
images, such as by subtracting one image from the other. If there is no entry or 
movement of objects in the monitored area, the projected pattern in the two images will 
substantially cancel out. However, if there is entry or movement of an object in the 
monitored area, part of the projected pattern shown in one image may be shifted or 
otherwise deformed relative to the pattern shown in the other image. 

Another method is to calculate a difference "gref ' between the value of the 
brightness levels corresponding to the light areas of the pattern (such as in a mask 
window), and the value of the brightness levels corresponding to the dark areas in the 
mask window of the reference image. A similar calculation may be made for the mask 
windows of the live image. Yet another method is to measure a correlation between each 
pixel and some neighboring pixels and/or a correlation between selected features, and 
then compare the correlation values. Other illustrative methods include extracting the 
moire spatial frequency and phase using a Fourier transform, unsharp masking, 
thresholding, contrast segmentation, filtering processing, skeletonization processing, 
multi-resolution analysis, deformable contour modeling, image clustering, morphology, 
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etc. These comparison methods are meant to be only illustrative, and that any suitable 
method may be used to compare the images or selected characteristics of the images, 
depending on the application. 

It is also contemplated that the reference and/or live images may be preprocessed 
before they are compared. For example, the reference image and/or live images may be 
provided to a filter that helps removes speckle, provides smoothing, changes overall 
intensity, or otherwise cleans-up the images. 

Step 80 determines if the differences identified in step 78 exceed the threshold 
value specified in step 70. If the differences exceed the threshold value, control is passed 
to step 82. Step 82 signals that an object is present in the monitored area. In some 
embodiments, an action is then taken, such as sounding an alarm, shutting down a 
machine, and/or providing some other alarm or action. If the differences do not exceed 
the threshold value, control is passed to step 84. Step 84 signals that an object is not 
present in the monitored area, and control is passed to step 86. 

Step 86 determines if an updated reference image is desirable. Under some 
circumstances, such as when the lighting conditions are dynamic in or around the 
monitored area, it may be advantageous to periodically update the reference image. If it 
is determined that an updated reference image is desirable, control is passed to step 74. 
Step 74 updates the reference image with the previous live image. Alternatively, a new 
reference image may be captured, if desired, so long as no objects have entered the 
monitored area. If it is determined that an updated reference image is not needed, control 
is passed to step 76, wherein a new live image is captured. 

Figure 8 is a flow diagram showing another illustrative method in accordance with 
the present invention. The illustrative method is entered at step 90, wherein a threshold 
value is selected. Control is then passed to step 92. Step 92 illuminates a monitored area 
with a first pattern. Control is then passed to step 94. Step 94 imposes a second pattern 
relative to the first pattern. Step 94 may, for example, illuminate the monitored area with 
the second pattern, or a grating may be placed between the monitored area and an image 
capture device, as desired. 
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Step 96 then captures a reference image of the monitored area. The reference 
image preferably contains only the static background of the monitored area, with no 
transitory objects present. In some embodiments, the reference image can be 
captured/updated upon manual initiation, as shown at 1 10. 

Step 98 captures a live image of the monitored area. Step 100 compares the 
reference image and the live image to determine selected differences. Step 102 
determines if the differences identified in step 100 exceed the threshold value specified in 
step 90. If the differences exceed the threshold value, control is passed to step 104. Step 
104 signals that an object is present in the monitored area. In some embodiments, an 
action is then taken, such as sounding an alarm, shutting down a machine, and/or 
providing some other alarm or action. If the differences do not exceed the threshold 
value, control is passed to step 106. Step 106 signals that an object is not present in the 
monitored area, and control is passed to step 108. 

Step 108 determines if an updated reference image is desirable. Under some 
circumstances, such as when the lighting conditions are dynamic in or around the 
monitored area, it may be advantageous to periodically update the reference image. If it 
is determined that an updated reference image is desirable, control is passed to step 96. 
Step 96 updates the reference image with the previous live image. Alternatively, a new 
reference image may be captured, if desired, so long as no objects have entered the 
monitored area. If it is determined that an updated reference image is not needed, control 
is passed to step 98, wherein a new live image is captured. 

Figure 9 is a flow diagram showing yet another illustrative method in accordance 
with the present invention. The illustrative method is entered at step 120, wherein a 
threshold value is selected. Control is then passed to step 122. Step 122 illuminates a 
monitored area with at least one pattern. Step 124 captures a reference image of two or 
more mask windows of the monitored area. Each mask window preferably corresponds 
to a sub-area or region within the monitored area. In some embodiments, the reference 
image can be captured/updated upon manual initiation, as shown at 140. 

Step 126 captures a live image of each mask window within the monitored area. 
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Then, for each mask window, step 128 compares the reference image and the live image 
to determine differences therebetween. In some embodiments, selected mask windows of 
the reference image and the live image are compared using different compare algorithms. 
Step 130 determines if any of the differences identified in step 128 exceed the threshold 
value specified in step 120. In some embodiments, each mask window or group of mask 
windows has a different threshold value. 

If any of the differences exceed the threshold value(s), control is passed to step 132. 
Step 132 signals that an object is present in the monitored area. In some embodiments, 
an action is then taken, such as sounding an alarm, shutting down a machine, and/or 
providing some other alarm or action. In some embodiments, the action taken may 
depend on which mask window detects an object. For example, for one mask window, 
the action may include sounding an alarm, while for another mask window the action 
may include shutting down a machine within the monitored area. In any event, if the 
differences do not exceed the threshold value(s), control is passed to step 134. Step 134 
signals that an object is not present in the monitored area, and control is passed to step 
136. 

Step 136 determines if an updated reference image is desirable. If it is determined 
that an updated reference image is desirable, control is passed to step 124. Step 124 
updates the reference image with the previous live image. Alternatively, a new reference 
image may be captured, if desired, so long as no objects have entered the monitored area. 
If it is determined that an updated reference image is not needed, control is passed to step 
126, wherein a new live image is captured of the mask windows of the monitored area. 

While the invention is susceptible to various modifications, and alternative forms, 
specific examples thereof have been shown in the drawings and are herein described in 
detail. It should be understood, however, that the invention is not to be limited to the 
particular forms or methods disclosed, but to the contrary, the invention is to cover all 
modifications, equivalents and alternatives falling within the spirit and scope of the 
appended claims. 
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